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ABSTRACT 



To help educators with accountability concerns, this guide 
provides a method for establishing and evaluating accountability systems in 
states, districts, and schools. Standards-based accountability refers to 
collecting and reporting information based on students' progress toward 
achieving established standards. To be useful, an accountability system must 
meet the varying information needs of parents, educators, policy-makers and 
the public. Ten policy issues can help establish this information base 
including: defining purposes and goals; establishing a design and 
implementing a process; deciding who will be held accountable and how 
progress will be measured; how performance and progress will be compared; at 
what levels data will be collected and reported; administrators should weigh 
the costs of implementing such a system; creating assessment with rewards, 
sanctions, and other incentives; helping the public understand results; 
supporting teachers, schools, and districts; and fine-tuning the system. Two 
appendices provide indicators of state accountability systems and a glossary 
of terms. (RJM) 
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EXECUTIVE SUMMARY 



The challenge facing American education is to help all 
students learn at higher levels than ever before. Setting 
high academic standards — clearly defined statements 
of what students are expected to know and be able to do 
at various points in their schooling — is a crucial part 
of meeting this challenge. Standards provide policy- 
makers, educators, parents and the public with the 
means to monitor, measure and continuously improve 
student achievement and school quality. 

With the use of content standards as the foundation for 
education reform, many state leaders are rethinking the 
structure, purpose and emphasis of their states’ account- 
ability systems. Among the major questions policy- 
makers must address are the following: 

• Who should be held accountable? For what? 

• How should student and/or school progress be mea- 
sured, compared and reported? 

• At what levels of the system will accountability data 
be collected, analyzed and reported? 

• What consequences will be attached to the results? 

• How can accountability systems be structured to sup- 
port greater flexibility? 

• How can public support and understanding of the 
accountability system be encouraged? 

• What is the state role in creating support structures 
for districts? 

Whalt Is AccoMmltatoiMlty? 

Accountability refers to the systematic collection, 
analysis and use of information to hold schools, educa- 
tors and others responsible for student performance. 
Standards-based accountability refers to collecting and 
reporting information based on student progress on 
achieving established standards. Elements of an 
accountability system include input measures (such as 
qualification of teachers or dollars spent on education), 
process measures (such as descriptions of how mathe- 
matics or science classes are taught) and outcome 



measures (such as student achievement measures or 
graduation rates). 




To be useful, an accountability system must meet the 
varying information needs of parents, educators, policy- 
makers and the public. To address these needs, state, 
district and school accountability systems should be 
developed in a coordinated fashion. The state account- 
ability system should monitor the progress of schools 
and districts and hold them accountable for helping stu- 
dents achieve high standards. District accountability 
systems should collect information related to state and 
district goals and standards and hold schools account- 
able for their performance targets. School accountabil- 
ity systems should focus on collecting achievement data 
related to school, district and state goals while provid- 
ing diagnostic information for improving student learn- 
ing and school improvement. 

Policy Issines im Besigmiiiig amd 
Implememltiinig AccoMmltatoilMy Systems 

POLICY ISSUE 1: Defining Purposes and Goals 

Accountability systems must be designed to achieve 
specific purposes. These purposes can include: 

• Monitoring and reporting school and student 
progress on achieving standards 

• Evaluating the performance of the education system 

• Holding schools and others accountable 

• Allocating resources 

• Certifying or promoting students 

• Improving student and school performance 

• Selecting or placing students 

• Planning staff development. 
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The challenge is to design a coordinated accountability 
system that emphasizes different purposes at different 
levels — state, district, school and classroom — based 
on the feasibility of data collection, data needs and 
information needs. 

POLICY ISSUE 2: Establishing a Design and 
Implementation Process 

States may choose to appoint a working group or create 
an independent panel charged with the design, imple- 
mentation and maintenance of the accountability sys- 
tem. This panel makes recommendations related to 
issues, such as how performance will be assessed and 
progress reported; how accountability results will be 
used to foster change and improvement; and what struc- 
tures will be put in place to build local capacity and sup- 
port school-level accountability. The design process 
usually begins by determining a set of principles and 
goals for the accountability system. 

POLICY ISSUE 3: Deciding Who Will Be Held 
Accountable and How Performance and Progress 
Will Be Measured 

Policymakers can choose from a number of different 
types of student achievement measures, including 
norm-referenced tests, criterion-referenced tests, cus- 
tomized norm-referenced tests, performance assess- 
ments or portfolios. Each has distinct advantages and 
disadvantages, and costs of administering each can vary 
widely. In a standards-based accountability system, stu- 
dent achievement measures that provide accurate 
assessments of student progress on standards should be 
the highest priority. Other indicators also should be col- 
lected to provide an accurate picture of the system and 
identify areas in need of improvement. 

POLICY ISSUE 4: Deciding How Performance and 
Progress Will Be Compared 

At the heart of any accountability system is the deter- 
mination of what constitutes satisfactory performance. 
Comparing the scores of students, schools or districts is 
a common way to make this determination. There are 
several approaches to the use of comparisons, 
including: 



• Comparing students or schools to absolute perfor- 
mance on achieving standards using proficiency 
levels 

• Comparing students’ or schools’ gain scores or 
progress toward performance targets 

• Comparing schools or students to “expected” scores. 

Most state accountability systems report student 
progress using proficiency levels on standards and on 
how well schools meet their performance targets. 

POLICY ISSUE 5: Deciding at What Levels To 
Collect and Report Data 

Accountability results can be collected and reported at 
the state, district, school or individual student level. 
The levels of reporting usually are determined by the 
purposes of the accountability system as well as cost 
considerations. Results most often are reported at the 
school level because most state accountability systems 
hold schools accountable for student results. School- 
level results, however, can be aggregated to create dis- 
trict- and state-level results. Some states report 
individual student scores, usually on measures that cer- 
tify or promote students, and other states only report 
district results. 

POLICY ISSUE 6: Weighing the Costs 

A number of design elements affect the costs of 
accountability systems and the student assessments on 
which they rely. These accountability system features 
tend to increase costs: custom-developed tests;, tasks or 
items that are “hand scored”; assessing a large number 
of items, grades or subject-matter areas; high stakes 
associated with test results; rewards for high-perform- 
ing schools; and the extensiveness of the set of indica- 
tors collected. 

POLICY ISSUE 7: Creating Rewards, Sanctions and 
Other Incentives 

Rewards and sanctions offer the potential for focusing 
teachers’ work, motivating school improvement efforts 
and improving teaching and learning. Six states have 
created accountability systems that use rewards and 
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sanctions to encourage schools and districts to increase 
student performance on standards. Rewards can be in 
the form of cash, recognition or increased flexibility. 
Sanctions usually are implemented on a continuum, 
starting with assistance and possibly ending with the 
closing of a school or a “state takeover” if adequate 
progress is not made. 

POLICY ISSUE 8: Helping the Public Understand 
Results 

How the public, policymakers and educators view — 
and whether they understand — the accountability sys- 
tem has much to do with its success. For parents, teach- 
ers and the public to see the accountability system as 
fair, they need to understand how the accountability 
system works, what led to specific rewards and sanc- 
tions being used, and why such actions are likely to be 
productive. Finally, educators and policymakers must 
work together to help parents and the public understand 
new reporting formats and how a standards-based 
accountability system differs from previous systems 
that may have used norm-referenced tests. 

POLICY ISSUE 9: Supporting Teachers, Schools 
and Districts 

State efforts to ensure accountability need to focus as 
much on creating the conditions for effective teaching 



as they do on holding schools accountable for results. 
For teachers to be effective, they need to have the 
knowledge and skills to teach effectively, helping all 
students meet high standards. States can play several 
different roles in providing support for educators, 
including creating standards for licensure/certification 
and professional development, organizing professional 
development centers, providing additional resources, 
developing reform networks or providing direct assis- 
tance to schools or districts. 

POLICY ISSUE 10: Fine-Tuning the System 

The success of an accountability system is determined 
not only by its design but also by the level of commit- 
ment policymakers demonstrate to its ongoing imple- 
mentation and refinement. The task force or panel 
charged with implementation of the program should 
conduct an annual survey of schools and/or districts to 
get feedback on the status of implementation and the 
system’s effectiveness; prepare an annual report to the 
legislative education committees and state board of edu- 
cation on the status of program implementation with 
recommendations for changes, if needed; and, conduct 
a longitudinal study of the impact the accountability 
system has on student performance. 
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INTRODUCTION 



** Improving student performance requires a clear picture of what you want to accomplish, a 
comprehensive measurement system to gauge progress and a commitment to act on the 
results to make appropriate changes. ” 

Colorado Governor Roy Romer 



The challenge facing American education is to help all 
students learn at higher levels than ever before. Setting 
high academic standards — clearly defined statements 
of what students are expected to know and be able to do 
at various points in their schooling — is a crucial part 
of meeting this challenge. Standards provide policy- 
makers, educators, parents and the public with the 
means to monitor, measure and continuously improve 
student achievement and school quality. 

States began to promote the use of standards in the late 
1980 s in response to mounting public concern over 
declines in student achievement, as measured by 
national and international assessments. This trend 
raised doubts about America’s ability to maintain the 
quality of its workforce and compete in the global mar- 
ketplace. The quality of America’s schools came to be 
seen as more important and less acceptable than ever 
before. And it was student performance — not simply 
“seat time” — that mattered. The consensus was that 
schools should be judged more by “outcomes” (student 
achievement) than by “inputs” (resources, facilities, 
number of advanced degrees among teachers and so 
forth). Standards would allow the education system to 
be judged by, and to be held more accountable for, 
results. 

With the use of content standards as the foundation for 
education reform, many state leaders are rethinking the 
structure, purpose and emphasis of their states’ 
accountability systems. Making the shift from a con- 
ventional, input-focused accountability system to one 
driven by standards entails a number of complex design 
and implementation issues. Among the major questions 
policymakers must address are the following: 

• Who should be held accountable? For what? 

• At what levels of the system (state, district, school, 
classroom) will accountability data be collected, ana- 
lyzed and reported? 




• How will student and/or school progress be mea- 
sured, compared, judged and reported? 

• Will — and, if so, to what extent — accountability 
data be used as a basis for rewarding or sanctioning 
schools or certifying students? 

• What are the costs and trade-offs of accountability 
design features? 

• How will the accountability system be structured to 
promote and support greater flexibility and authority 
at the local level? 

• How can understanding of and support for standards- 
based accountability be strengthened among educa- 
tors, parents and the public at large? 

• How will the appropriate state role in supporting 
school and district improvement be determined? 

• How should a coordinated accountability system that 
provides information for policy decisions at all levels 
be designed? 

Using This Gniidle 

This guide is designed to heighten policymakers’ under- 
standing of these choices and challenges. It describes 
the various ways standards-based accountability sys- 
tems can be designed and used, and how they can con- 
tribute to state and local education improvement. The 
underlying premises of accountability systems used in 
this guide are: 

• Academic standards should provide the basis for 
evaluating student performance and progress. 

• Schools should be held accountable for student 
achievement and student progress toward standards. 
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Standards-based accountability 
systems . . . are a vital tool in state 
and local education improvement 
efforts. 



• Schools must be provided local authority and flexi- 
bility to use various instructional approaches to help 
students achieve standards. 

• The most useful accountability system is one focused 
on motivating and supporting higher performance 
and continuous improvement at all levels of the edu- 
cation system — state, district, school and class- 
room. 

The first section of this guide examines in a question- 
and-answer format the basic definitions, elements and 
limitations of standards-based accountability systems. 
The second section outlines 10 policy issues in the 
design and implementation of such systems, and pro- 
vides a look at various approaches, design options and 
trade-offs, and support structures available to policy- 
makers. A glossary (see Appendix B) defines terms 
used in the report. 

Developing standards-based accountability systems is a 
difficult and complex process, but such systems are a 
vital tool in state and local education improvement 
efforts. When designed thoughtfully and implemented 
successfully, accountability systems can help more stu- 
dents achieve at higher levels and help improve the 
overall efficiency and effectiveness of public education. 
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UNDERSTANDING STANDARDS-BASED 
ACCOUNTABILITY 



Whall; Is AccoEirtability? 

Accountability refers to the systematic collection, 
analysis and use of information to hold schools, educa- 
tors and others responsible for the performance of stu- 
dents and the education system. 

Accountability is most effective when it does the 
following: 

• Links authority and adequate resources to responsi- 
bility 

• Defines clear lines of responsibility and mutual 
obligations 

• Involves fair and adequate assessment against 
agreed-upon goals 

• Involves appropriate consequences (both rewards 
and sanctions) for observed performance. 

■ How Is AccoEimtalbiMty Bifffereimt 
from Assessmeell;? 

The concepts of assessment and accountability some- 
times are confused, since student test results are usually 
the primary element of state indicator systems used to 
hold schools, or students, accountable. In this docu- 
ment, accountability systems are defined as: the system- 
atic collection of input, process and outcome data, as 
well as the use of these data, to make decisions about 
the effectiveness of schools, districts or states. 
Therefore, accountability systems differ from assess- 
ment both in terms of the breadth of information col- 
lected and the manner in which the information is used 
in making decisions. 

Whail; Is Standards-Based 




The focus of a standards-based accountability system is 
to measure success against clearly defined standards. A 
standards-based accountability system focuses attention 
on student learning rather than on compliance with rules 
and regulations. It allows states to move away from reg- 
ulating schools based on inputs (the number of books in 

the library and the proportion of certified staff, for 

-X<- 



example) and move toward a model of steering by 
results — using rewards, sanctions and assistance to 
move schools toward higher levels of performance.^ 

To assess student learning, state or local leaders first 
establish standards that define what students should 
know and be able to do at various points in their school- 
ing. In some states, such as Delaware, standards were 
determined by a joint effort of state and local leaders 
and are expected to be used by every school district. In 
other states, such as Colorado, the state developed 
model content standards. Districts are expected either to 
adopt the state standards or develop their own standards 
that “meet or exceed” the state standards. In another 
model exemplified by Iowa, standards are determined 
by each school district. 

Performance standards — which provide concrete 
examples and definitions of how well students must 
learn the material represented by content standards — 
then are used to create performance levels that define 
students’ demonstrated proficiency_at various_points_as 
they progress toward meeting a standard. 

In a standards-based system, states focus not so much 
on how, but on whether, schools are helping students 
achieve the standards. The emphasis is on holding 
schools accountable for student performance rather than 
on the methods schools use to achieve results. 

Whail; Are tlhe Basic Elements 

off a Standards-Based AccoEntability 




Accountability systems typically collect a wide variety 
of information about districts or schools that can be 
characterized as input, process and outcome measures. 

• Input measures include information such as the qual- 
ifications of teachers, the dollars spent on education, 
and the numbers and types of students being served. 
In a standards-based accountability system, inputs 
can be used as background information to evaluate 
gains in student achievement and better understand 
the environment in which schools are operating. 
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• Process measures describe the types of programs 
offered in schools. For example, they might include 
descriptions of how science or mathematics classes 
are taught, what professional growth opportunities 
are offered to teachers and what types of special-edu- 
cation programs are available for students with dis- 
abilities. 

• Outcome measures are used to describe the achieve- 
ment and accomplishments of students, including 
grades, test scores and the number of students gradu- 
ating from high school, entering college or becoming 
employed. 

Standards-based accountability systems focus on stu- 
dent performance (outcomes), with input and process 
measures used to help explain or interpret the outcomes. 
States are beginning to use and collect data on multiple 
measures — and even accept alternative measures — 
when the results are used to make important decisions 
related to students. Since a variety of types of informa- 
tion are available about schools, the challenge is for 
policymakers and educators to select from among the 
numerous available data elements those that are most 
useful in creating a clear, coherent picture of the out- 
comes of a school, district or state. 

Mow Cam Stamdards-Based 
Accommlalbility Be Used To Promote 
Improvememt at Varioms Levels 
off tlie Edmcatiom System? 

To be useful, an accountability system must meet the 
varying information needs of parents, educators, state 
and local policymakers, and the public.^ 

• Parents want to know what their child knows and can 
do, while a teacher is more concerned with identify- 
ing a student’s strengths and what instructional chal- 
lenges are needed. 

• A school principal needs to know if student achieve- 
ment is comparable to that in similar schools and how 
well students are achieving state and local standards. 



• District leaders may be more concerned about 
whether the achievement needs are greater in mathe- 
matics than reading, so that additional resources can 
be allocated for mathematics instruction. Or they 
may be concerned that several schools are not meet- 
ing their performance targets. 

• At the state level, concern is often focused on 
whether there is equity in school programs, whether 
differences in student performance are due to the lack 
of resources in some schools, and whether students 
have the knowledge and skills to help the state 
remain economically competitive. 

To address these various information needs, account- 
ability systems should be developed in a coordinated 
fashion across the multiple levels of the education sys- 
tem. Using assessments as an example. Figure 1 (page 
5) shows how the need for information on specific stu- 
dents expands as assessments move from the state level 
to schools and classrooms. 

The implications of these information needs for the 
design of state, district, school and classroom assess- 
ments include the following: 

• The state accountability system should focus on mon- 
itoring student progress on achieving state standards, 
and holding schools and districts accountable for 
that progress. State assessments should be aligned 
with state content standards to provide targets for 
instruction. States may promote the administration 
and scoring of performance tasks at the local level 
that are reported to the state and used as part of the 
state accountability data. 

• The district accountability system should focus on 
collecting information related to district goals and 
standards while holding schools accountable for 
their performance targets. Districts may choose to 
assess students in grades or academic areas not cov- 
ered by the state assessment. More performance 
tasks can be administered and scored by teachers to 
guide their instruction and provide information on 
individual students. District-level data should give 
local policymakers guidance on which school pro- 
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grams are most effective, identify areas where 
resources need to be reallocated to support improve- 
ments, and provide comparisons among schools that 
are shared with parents and community members. 

• School-level accountability should focus on the col- 
lection of achievement data related to school goals 
while providing diagnostic information for improv- 
ing student learning and facilitating student 
placement. 



• At the classroom level, more direct and complex 
assessments, such as performance tasks, portfolios or 
mastery exhibitions should be used for student diag- 
nosis and program improvement. 

Table 1 (pages 6 and 7) suggests some purposes 
and design features of a coordinated, multi-level 
accountability system. 



Figure 1 



RELATIONSHIP BETWEEN LEVEL OF ASSESSMENTS AND KNOWLEDGE 

OF SPECIFIC STUDENTS 




National State District Classroom 

Assessments Assessments Assessments Assessments 



Source: A Comprehensive Guide to Designing Standards-Based Districts, Schools, and Classrooms by Robert 
J. Marzano and John S. Kendall, p. 89. Reprinted with permission from the Mid-Continent Regional Educational 
Laboratory (Aurora, CO). 
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Table 1 

PURPOSES, DESIGN FEATURES AND USES OF ACCOUNTABILITY SYSTEMS 
AT THE STATE, DISTRICT, SCHOOL AND CLASSROOM LEVELS 



Level 



Purposes of Accountability 
System 



Primary Design Features 
of an Accountability System 
Based on Purposes 



Accomplishing the Purposes: 
Uses of Accountability 
Information 



State 



• Monitoring system progress 

• Identifying the relative perfor- 
mance of one system to another 

• Holding districts and schools 
accountable 

• Providing targets for instruction 

• Providing information for program 
improvement 

• Allocating resources 

• Certifying students 



• Indicators could include informa- 
tion on expenditures, district or 
school demographics, program 
quality 

• Assessment is aligned to state 
standards and assesses only core 
academic standards 

• At least three grades are assessed 
— one in elementary school, one 
in middle school and one in high 
school 

• Assessment item formats are pri- 
marily selected-response with 
some short-answer, extended- 
response and performance tasks 
to assess more complex standards 

• If accountability system is “high 
stakes," then assessment meets 
high technical and legal criteria 



• Providing rewards and sanctions 
for high- and low-performing dis- 
tricts or schools 

• Revising state policy to Improve 
adequacy and equity 

• Reallocating resources to where 
they are most needed 

• Targeting assistance to low-per- 
forming schools 

• Providing information to the public 
on the status of schools and the 
education system 

• Endorsing diplomas or providing 
certificates of mastery 

• Focusing teachers’ attention on 
standards 



District 



• Monitoring district progress 

• Improving student performance 

• Holding schools accountable 

• Providing targets for instruction 

• Allocating resources 

• Evaluating program effectiveness 

• Providing targets for instruction 



• Indicators collect information on 
school demographics, school-level 
expenditures, school program, 
teacher quality and parent 
satisfaction 

• Assessment is aligned to district 
standards and may be adminis- 
tered at every grade or at grades 
not assessed by the state account- 
ability system 

• District assessment results are 
reported to the community and 
may be included in the state 
assessment results 

• Assessment item formats include 
selected-response and can include 
more short-answer, extended 
response, performance tasks and 
portfolios to assess more complex 
skills and variations in school 
designs while meeting technical 
accuracy 



• Providing rewards and sanctions 
for high- and low-performing 
schools (Including determining if 
schools meet their performance 
contract agreements) 

• Reallocating resources to schools 
that need additional assistance 

• Identifying effective programs such 
as whole-school designs 

• Providing comparisons with other 
(similar) schools in the district 

• Reporting school-level data to the 
public to promote conversations 
related to school designs and 
school improvement; creating a 
consumer guide to district schools 




o 
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Table 1 (Continued) 

PURPOSES, DESIGN FEATURES AND USES OF ACCOUNTABILITY SYSTEMS 
AT THE STATE, DISTRICT, SCHOOL AND CLASSROOM LEVELS 



Level 



Purposes of Accountability 
System 



Primary Design Features 
of an Accountability System 
Based on Purposes 



Accomplishing the Purposes: 
Uses of Accountability 
Information 



School 



' Evaluating program effectiveness 
' Placing students 
' Certifying students 
' Improving student performance 



Indicators could include commu- 
nity characteristics, student demo- 
graphics, attendance rates, 
graduation/completion rates, track- 
ing post-school student perfor- 
mance, school-level inputs such 
as resources allocated to profes- 
sional development, instructional 
materials, and degree of imple- 
mentation of school program 

' Assessments are aligned with 
school goals and school program 

' Assessment item formats can 
include selected-response, short- 
answer, extended-response, per- 
formance tasks, portfolios and 
mastery exhibitions 

• Assessment results are available 
in time to make improvements 



• Input for school’s strategic plan- 
ning process 

• Evaluating how well schools meet 
their goals 

• Evaluating effectiveness of school 
design for student population and 
community needs 

• Input for reallocating resources 
within the school 

' Basis for communitywide conver- 
sations on school improvement 

' Gauging parent satisfaction with 
the school program 

' Providing information for district’s 
consumer guide to schools 



Classroom • Student diagnosis 

• Improving student performance 



' Assessments are aligned with 
school program and are appropri- 
ate for the grade or student group 

• Assessment results are immedi- 
ately available to use for student 
diagnosis to improve student 
performance 



• Providing information on individual 
students for communication with 
parents 

• Providing information to place or 
select students 

• Providing information on effective- 
ness of classroom teaching and 
learning 
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A flexible system recognizes that, 
when it comes to educating students^ 
one size does not fit all. 



Why Is Flexibility a Critical Compomeet 
off Stamdla]rdls=Basedl Accouimtability 




If educators and others in the system are to be held 
accountable for student performance, they must have 
the authority and responsibility to meet those goals. 
Because students have different backgrounds and learn- 
ing styles, schools need to take different approaches to 
meet their learning needs. To provide this needed flexi- 
bility, many educators and policymakers are turning to 
“whole-school designs.” Exemplified by the New 
American Schools (NAS) designs, whole-school 
designs provide different blueprints, approaches or 
philosophies to help schools and communities work 
together to help all students meet high standards. 

Flexibility, by promoting choices, gives schools more 
autonomy to be responsive to parents’ wishes and stu- 
dent needs, gives teachers and administrators a stronger 
sense of purpose and responsibility, creates models of 
innovation and encourages schools to use their 
resources more efficiently. A flexible system recog- 
nizes that, when it comes to educating students, one 
size does not fit all. 

A flexible system requires an operating environment 
that maximizes schools’ ability to help all students 
reach high levels of achievement. First, all schools 
should have clearly defined standards that reflect high 
expectations for students. Multiple forms of assessment 
should be used to measure student progress toward the 
standards. Teachers should be provided assistance to 
help them strengthen instruction, align curriculum with 
higher standards, build in a variety of ongoing assess- 
ments to develop a rich picture of student learning and 
adjust their practice to meet individual students’ needs. 
Finally, schools need the authority to decide how to 
deploy all available resources — money, staff and time 
— to help students reach high standards.^ If schools are 
to be held responsible for student learning, they need 
the authority to make decisions to accomplish this. 

When well-designed accountability systems are cou- 
pled with other policy changes, such as standards-based 
reform, site-based management and budgeting, public 
school choice, charter schools and reform networks, 

vi t 7 
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schools are given more authority to produce results, yet 
are held accountable by parents and community mem- 
bers. Schools also are held accountable by local school 
boards, which may close schools that do not meet 
agreed-upon performance goals. 

What Are LnmntatnoES of AccoeEtatonlnty 




The biggest limitations of accountability systems ai’e 
the quality and quantity of the information reported, and 
the alignment of the data to the particular school and 
district. Are the data current? If the data are one or more 
years old, they may not be pertinent to the school or dis- 
trict at the current time. Are the data elements related to 
the school? For example, a 7th- grade score may reflect 
more about the elementary schools feeding the middle 
school than the middle school itself. 

The data elements also may be limited. For example, 
student performance is more important in an account- 
ability system than data about inputs (e.g., expenditures 
or staffing). It is not uncommon, however, for the latter 
to be more prevalent than the former types of informa- 
tion. 

There are other limitations to accountability systems 
worth noting. 

• First, much of the information reported as indicators 
typically is obtained from external sources. For 
example, college-entrance test scores, state assess- 
ment scores, and financial and staffing data may be 
reported by the state or by national organizations. 
Hence, they may not fit well with the local school 
system and may contain errors as the data are col- 
lected, edited and reported by others. 

Other information that may be available but is not 
reported might change some of the observations 
reported or decisions made. For example, information 
that classroom teachers collect, such as portfolio data 
or their own tests, might add another dimension to the 
reporting of student accountability information. Since 
these data are rarely collected and reported in state 
accountability systems, however, these potentially 
valuable insights are not reported, and the picture of 




student performance may be somewhat misleading. 
Are there other sources of information that can or 
should be used? 

• Second, some important outcomes of schooling are 
not easily measured by the indicators selected for 
reporting. The most valued outcomes of schooling 
(e.g., gainful employment, successful parenting, sat- 
isfying life) are not easily measured while students 
are in school or even later in life. Hence, things such 
as grades and coursetaking are substituted for these 
more important longer-term outcomes. 

• Third, achievement data and other indicators are 
most often cross-sectional in nature. This means the 
data presented at different grades for a particular year 
come from different groups of students. Are changes 
observed due to differences in students or differences 
in the education system? 

• Fourth, none of the explanatory factors that may be 
associated or “correlated” with student achievement 
necessarily account for student performance. For 
example, socioeconomic status and achievement are 
often correlated, yet poverty does not cause low 
achievement. Since the effective schools research 
abounds with high-poverty schools with high 
achievement, there are clearly other factors, such as 
teacher expectations or parental involvement in edu- 
cation, that are at work. The fact that two variables 
are statistically correlated does not mean that one 
causes the other, and such causal inferences should 
be avoided. 

Users of accountability system information should be 
cautious before putting forth explanations for the 
causes for student performance. Is one variable that 
is correlated with another actually caused by that 
variable? 

• Finally, most indicators provided in accountability 
reports contain only one to three years of data. Where 
possible, more data should be provided; trend analy- 
sis should not be carried out with fewer than three 
years of data. If more data were available, would the 
same trends in performance be observed? 
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POLICY ISSUES 



There is no single “right” way to design and develop 
standards-based accountability systems. A state’s 
accountability system will, and must, reflect the social 
and political environment within the state, as well as the 
state’s goals and objectives.^ The experience of several 
states that have developed or are in the process of 
designing standards-based accountability systems, 
however, suggests a number of critical steps in the suc- 
cessful design and implementation of such systems. 



There is no single “right” way to 
design and develop standards-based 
accountability systems. 



POLICY ISSUE 1; 

BelKeirng Perposes aed Goals 

Accountability systems can serve several different pur- 
poses. An accountability system that attempts to per- 
form too many functions, however, inevitably will do 
none of them well. Therefore, it is important to identify 
desired purposes and to design an accountability system 
to serve those purposes effectively. One way to serve 
multiple purposes is to create a coordinated account- 
ability system where different purposes are identified 
and met at different levels of the education system. 

In some cases, an accountability system designed for 
one purpose cannot be used effectively for another pur- 
pose. For example, using a traditional “off-the-shelf’ 
norm-referenced test in an accountability system, 
although providing student percentile rankings, may 
not be effective in assessing student progress toward 
state or local content standards. Likewise, limiting data 
collection in an accountability system to district- level 
samples may not provide enough information for 
rewards and sanctions to be applied to individual 
schools. The worst-case scenario is for policymakers to 
establish the structure of a system without first deter- 
mining the purposes. To do so may mean that multiple, 
sometimes conflicting, purposes (and messages to edu- 
cators) may emerge, and that users of the accountability 
system may be unclear about how the information 
resulting from the system is to be used. 



In a standards-based accountability system, the most 
obvious purpose is to monitor^ evaluate and publicly 
report the progress of students, schools and districts 
toward achievement of content standards and other 
established goals. 



ERIC 
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Other purposes can include: 

® To provide information for policy decisions. 
Accountability data can be used to evaluate or for- 
mulate policy, especially policies focused on equity 
and resource allocation. 

© To provide information for program improvement. 
An accountability system can include the collection 
of data on multiple, related indicators and report the 
results for classrooms and schools. The results can 
help identify areas of strength and weakness, either 
on an absolute or relative basis, so schools will know 
which programs or services need improvement. 

© To evaluate the performance of the education system 
relative to other systems. Sometimes, accountability 
systems are designed to permit external comparisons 
of state, district or school performance with the per- 
formance of systems in other states or countries. 

© To hold schools and/or districts accountable. 
Standards-based accountability systems collect 
information on students’ proficiency on standards, 
allowing school and district scores and reports to be 
analyzed and compared. 

• To allocate resources. Standards-based accountabil- 
ity systems can be used to help allocate or target 
resources, such as providing rewards for high-per- 
forming schools and assistance for low-performing 
schools. 

® To certify or promote students. Some states use 
accountability systems to retain students or promote 
them to the next level (such as from elementary 
school to middle school) or to certify students for 
graduation. When accountability systems are used 
for high-stakes purposes, the measures relied on 
must be legally defensible and of high technical 
quality. 

• To improve individual student performance. The 
assessment portion of the accountability system can 
be used to provide feedback to individual students on 
their strengths and weaknesses and provide informa- 
tion on how learning might be improved. 



An accountability system that 
attempts to perform too many 
functions, however, inevitably will 
do none of them well. 
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The challenge is to design a coordi- 
nated accountability system that 
emphasizes different purposes at 
different levels. 



• To select or place students. The assessment portion 
of the accountability system can be used to select or 
place students in various programs based on their 
needs and level of achievement. 

• To plan staff development. School accountability 
results can be used to identify curricular or instruc- 
tional weaknesses for targeted professional develop- 
ment. 

The challenge is to design a coordinated accountability 
system that emphasizes different purposes at different 
levels — state, district, school and classroom — based 
on the feasibility of data collection, data needs and 
information uses. 

POLICY ISSUE 2 % 

Establishing a Design and 
ImpSementation Process 

Once the central focus and purpose of the accountabil- 
ity system has been defined, a number of crucial issues 
— some involving policy choices, others involving 
technical issues and questions — must be addressed. 
These issues include the following: 

• What will the unit of accountability be (student, 
school, district — or all three)? 

• How will performance and progress be measured, 
compared and reported? 

• How will accountability data be used to foster 
change and improvement? 

• What role will local school districts, the state depart- 
ment of education and other agencies play in the 
accountability process? 

Addressing these and other important questions in a 
coordinated, coherent fashion can require the creation 
of an independent panel charged with the design, imple- 
mentation and maintenance of the accountability sys- 
tem. The establishment of such a panel, whose 
composition and responsibilities may be set in statute or 
by state board of education rule, sends a message of 
objectivity, fairness and importance. 
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Resources for the design and development process 
should be identified at the outset. External expertise 
may be required, although a great deal of support can be 
provided by the state department of education, univer- 
sities and practitioners within the state, providing addi- 
tional opportunities for buy-in on the part of various 
groups and individuals. Task forces and ad hoc 
committees in such areas as assessment, professional 
development, collective bargaining, technology, post- 
secondary articulation and parent involvement can be 
used to extend the accountability panel’s resources. 

One of the first tasks a panel has is to create principles 
of an effective accountability system. This establishes a 
frame of reference for policy discussions and provides 
direction in establishing a knowledge base among the 
panel members. The next task is to translate these prin- 
ciples into purposes, goals and the design of an 
accountability system. 

In addition, a public information process is needed to 
include the media, as well as the legislative and execu- 
tive branches, schools, districts and the general public. 
Regular reports on the progress of design and imple- 
mentation can help minimize misinformation. Hearings 
and other opportunities for public comment during the 
design phase also can help add to the credibility and 
coherence of the process. 

POLICY ISSUE 3: 

DecMimg Wlho Will Be Held Accoe]mtt= 
able and How Peirfformamce and 
Progress Will Be Measured 

Most accountability systems focus on holding students 
and schools accountable. But, states are beginning to 
expand accountability systems to articulate expecta- 
tions for more participants in the education process, 
assess their performance and hold them accountable. 

For example, Delaware’s new accountability system 
(see sidebar) plans to hold teachers and administrators 
accountable by requiring them to meet professional 
teaching and administrator standards, respectively. 
These standards include program approval, initial licen- 
sure, induction process, evaluation and recertification. 






DELAWARE’S PRINCIPLES OF AN 
EFFECTIVE ACCOUNTABILITY SYSTEM 



• Accountability should be integrated with profes- 
sional development so that all students have the 
opportunity to succeed. 

• Accountability should be tied to progress toward 
the state's academic standards, with students and 
schools expected to perform at agreed-upon high 
levels. 

• Accountability should be linked to authority. 

• To the extent possible and practical, all system par- 
ticipants — students, parents, educators, schools, 
business and the community — should be account- 
able in some way. 

• Performance should be linked to consequences for 
individuals as well as schools, districts and the 
state. Schools whose students meet or exceed the 
standards should be rewarded. Those that need help 
should receive it. Those that persistently or dramat- 
ically fail should be penalized. 

• Progress toward student and professional standards 
should be measured with technically adequate and 
fair assessment tools. 

• The accountability system should be easily under- 
stood by, perceived as fair by and make good sense 
to the public and educators. 

• An accountability system must include help for stu- 
dents having difficulty meeting the requirements. 

• Accountability must be supported by an information 
system and analytic capacity that ensure improved 
policy, programs and practices. 

Source: The Missing Link: Connecting Professional 

Development with Accountability To Improve 

Student Learning in Delaware, 1997. 
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Other states, such as Tennessee, hold teachers account- 
able by measuring the gain in student achievement dur- 
ing the academic year. Delaware also plans to hold 
department of education staff accountable based on 
employee performance, program evaluations, customer 
satisfaction surveys and annual overall gains of the edu- 
cation system. See Table 2 on page 16 for state account- 
ability examples of who is being held accountable and 
for what. 

Student Achievement Measures 

Nearly every state uses one or more assessments to col- 
lect information about student achievement.^ Among 
the types of tests used in state accountability systems 
are the following: 

• Norm~referenced tests. These tests are commercially 
available instruments that provide comparisons 
between individual student or school scores and 
those of a nationally representative norm group. 
Norm-referenced tests are inexpensive to purchase 
and score, provide national comparisons and are eas- 
ier for parents to understand. The disadvantages are 
that they rarely provide an exact match to a state or 
district’s content standards, and historically have not 
included performance tasks to drive more complex 
learning and instruction. 

• Criterion-referenced tests. These are typically cus- 
tom-developed assessments, based on a state’s con- 
tent standards (although credible “off-the-shelf’ tests 
aligned to standards developed at the national level 
are beginning to emerge). Selected-response and con- 
structed-response items are used, and scores are 
reported relative to a state’s performance standards. 
The advantage is that custom-developed tests are 
aligned closely with state and district standards; the 
disadvantage is that there are more costs associated 
with custom development than an off-the-shelf test. 

• Customized norm-referenced tests. Occasionally, 
assessment programs use a version of a norm-refer- 
enced test to maintain the national comparison 
advantages of the off-the-shelf test, but customize it 






to fit the particular state or local content standards. 
This approach shares the disadvantage of higher test- 
development costs with criterion-referenced tests, but 
allows the instruments to be aligned more closely 
with state and local standards than conventional 
norm-referenced tests. Another disadvantage is the 
frequent lack of “direct” or performance measures. 

• Performance assessments. Performance assessments, 
in which students actually produce work (an essay, a 
laboratory experiment, a painting), have the advan- 
tage of being closely linked to curriculum and con- 
tent standards. By connecting what is taught and 
what is tested, these assessments also may influence 
what teachers teach. They are, however, more expen- 
sive to develop, administer and score. 

• Portfolios. Portfolio assessment, in which examples 
of student work are compiled and evaluated over 
time, is another way to collect information about stu- 
dents. Some states, including Vermont and Kentucky, 
are using student portfolios as part of their state 
assessment system. Portfolios have the advantage of 
providing longitudinal data showing student growth 
over time. Their primary disadvantages are the time 
and resources needed to collect the information and 
to evaluate it fairly and reliably. If portfolios are used 
in high-stakes situations, care must be taken to ensure 
that scorers have been well-trained. 

Although states may use any of the tests described 
above, it should be noted that standards-based assess- 
ments (all of the types of tests mentioned above except 
norm-referenced tests) have several advantages over 
their traditional counterparts.’ These advantages include 
the following: 

• Standards-based assessments are closely linked to 
content and curriculum standards. By connecting 
what is taught and what is tested, assessments influ- 
ence what teachers teach. Standards-based assess- 
ments compare student accomplishment to 
preestablished goals, rather than to the performance 
of other students. 
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o They can incorporate new forms of assessment Such 
assessments can require students to undertake 
“authentic” tasks, such as writing an essay or con- 
ducting a hands-on science experiment. Such assess- 
ments capture a broader range of complex thinking 
and problem-solving skills. 

o They model tasks for teachers' professional develop- 
ment Performance tasks serve as examples teachers 
can use in their classrooms. Teachers’ scoring of per- 
formance assessments also provides an opportunity 
to discuss standards and performance expectations 
for students. 

It is possible for states to include multiple types of 
assessments in an accountability system. Maryland, for 
example, uses criterion-referenced tests and perfor- 
mance tasks in its primary state assessment, but offers 
districts the option of using norm-referenced tests on a 
limited basis to provide comparisons to national norms. 

In addition to student assessments, accountability sys- 
tems typically include a broader set of indicators. These 
indicators are usually reported in school report cards or 
school profiles. Indicators used by the states are catego- 
rized in Appendix A.^ 

Some indicators, such as student and family character- 
istics, may explain differences in performance between 
schools or districts. Other indicators, such as curriculum 
or advanced courses offered, are considered to be under 
the direct control of teachers, administrators or local 
school boards and may indicate the aspirational level of 
students or the school. Both types of indicators should 
be included in an accountability system. 

The challenge for policymakers is to identify and mea- 
sure the outputs that are most valued (rather than just 
those most available), and to collect information on the 
inputs and processes needed to provide an accurate pic- 
ture of the system and identify areas in need of 
improvement. 



POLICY ISSUE 4; 

Bedding How Perffoimniaiiice and 
Progress Will Be Compared 

At the heart of any accountability system is the deter- 
mination of what constitutes satisfactory performance. 
Comparing the scores of students, schools or districts is 
a common way to make this determination. There are 
several approaches to the use of comparisons: 

• Comparing students or schools to absolute perfor- 
mance, States frequently report student- or school- 
level data in terms of the percentage of students who 
meet or exceed a proficiency level on each standard. 
For example, the scores of 8th graders on a mathe- 
matics computational test might be reported as 25% 
“partially proficient,” 65% “proficient” and 10% 
“advanced.” Reporting absolute scores sends a mes- 
sage that the state’s goal is to have all students, 
regardless of background characteristics, perform at 
high levels. 

© Comparing students' or schools' . gain scores. Gain 
scores can be determined in several ways. Using a 
longitudinal data-collection approach, differences 
between scores of the same individual students are 
compared as they move through different grade lev- 
els. For example, a student’s score on a mathematics 
test taken in 7th grade is compared with the same stu- 
dent’s score on an equivalent test taken in 8th grade. 
School scores are the average gain made by matched 
students in a school. 

Longitudinal data collection is more expensive because 
the same students must be identified and tested, and 
results matched and compared. The advantage, how- 
ever, is that longitudinal data indicate the “value added” 
of the schooling experience. Tennessee’s assessment 
program uses this approach. 

Gain scores also can be computed using cross-sectional 
data-collection approaches. For example, the percent of 
proficient 8th-grade students in a given school one year 
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Table 2 

WHO SHOULD BE HELD ACCOUNTABLE? FOR WHAT? USING WHAT MEASURES? 
WITH WHAT SUPPORT? SOME POLICY OPTIONS 



Who Should 
Be Held 
Accountable? 



For What? 



Using What 
Measures? 



With What 
Consequences? 



Needing What 
Supports? 



Legislature • Adequate and equitable 

school funding 

• Policies aligned to 
standards 

• Legislation for account- 
ability system 



• Report card of voting 
records 

• Policy review 



• Constitutionality of funding 
formula 

• Re-election 



» Adequate state resources 
for education 
» Policy options to support 
standards 



State Board 
and State 
Department 
of Education 



* Creation of an 
accountability system and 
data collection 

* Standards 

► Public reporting of district 
and/or school-level 
accountability results 
‘ Statewide student 
achievement gains 
‘ Identification of low 
performing schools 
‘ Effective technical assis- 
tance to schools/districts 



- Student achievement 
measures 

- Other indicators 

> Customer satisfaction 
surveys 

> Program evaluations 
Employee performance 



> Election/appointment 
process 

> Board reconstitution 

> Legislative action 

> Performance-based com- 
pensation of employees 

> Probation, reassignment 
or dismissal of staff 



► Adequate funding of 
accountability program 
* Student achievement 
measures that are of high 
technical quality and 
aligned to standards 
‘ Adequate resources and 
staff expertise for techni- 
cal assistance to schools 
and districts 
‘ Mechanism to use 
accountability results for 
system improvement 
‘ An engaged public 



Teacher • Programs meet teacher 

Education and administrator program 

approval guidelines 

• Programs meet 
NCATE/NASDTEC 
standards 

• Teacher education appli- 
cants meet K-12 profi- 
ciency standards 

• Teacher education gradu- 
ates meet professional 
teaching standards 



► Student competency 
measures 

► NCATE/NASDTEC 
accreditation 

‘ Measures of professional 
teaching standards 



‘ Loss of teacher education 
program accreditation or 
approval 



► Redesign of teacher edu- 
cation courses to support 
standards- based system 

► Course instruction that 
models diverse instruc- 
tional strategies 



Local School • Meeting professional stan- 
Boards dards for school boards 

• Creating local policies that 
support a standards- 
based system 

• Creating districtwide 
improvement plan 

• Meeting district perfor- 
mance targets 

• Conducting public hear- 
ings on accountability 
results 



‘ Measures of board pro- 
fessional standards 
‘ Local policy review 
‘ Measures of meeting per- 
formance targets 
‘ District quality reviews 



‘ Election/appointment 
process 

‘ Board reconstitution 



• Adequate state policy 
framework 

• Adequate state and local 
resources 

• Central office staff able to 
implement programs and 
policies 

• Qualified teachers 



o 
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Table 2 (Continued) 

WHO SHOULD BE HELD ACCOUNTABLE? FOR WHAT? USING WHAT MEASURES? 
WITH WHAT SUPPORT? SOME POLICY OPTIONS 


Who Should 
Be Held 
Accountable? 


For What? 


Using What 
Measures? 


With What 
Consequences? 


Needing What 
Supports? 


Districts/ 

Schools 


• Student proficiency on 
academic standards 

• Meeting performance 
targets 

• Adequate inputs 

• Adequate processes 

• Meeting rules and 
regulations of state 
board 

• District and school 
improvement plans 


• Student achievement 
measures 

• Measures of indicators 
such as dropout, atten- 
dance and graduation 
rates 

• Accreditation measures 

• Program review 


• Public reporting of results 

• Rewards 

• Deregulation 

• Sanctions 

• Required assistance 


• Adequate resources and 
authority 

• Local policy framework 
that supports standards 
implementation 

• Curricula aligned to stan- 
dards 

• Access to technical assis- 
tance and school designs 

• School/community 
process to set goals, 
review, act on results 


Principals/ 

Administrators 


• Student proficiency on 
academic standards 

• Meeting professional 
administrator standards 

• Administrator evaluation 


• Student achievement 
measures 

• Measures of other indica- 
tors such as school and 
district goals 

• Administrator certification 
measures 

• Professional review 


• Initial certification and 
evaluation 

• Licensure or certification 

• Recertification 

• Salary bonuses 

• Remediation and support 

• Probation, reassignment 
or dismissal 


• Professional development 
opportunities 

• Adequate resources and 
authority 

• Access to technical assis- 
tance and school designs 


Teachers 


• Student proficiency on 
academic standards 

• Professional teaching 
standards 

• Teacher attendance 


• Student achievement 
measures 

• Measures of other indi- 
cators 

• Teacher certification 
measures such as Praxis 
or INTASC measures 


• Teacher evaluation tied 
to student achievement 

• Recertification 

• Differentiated staffing 

roles ’ * ' 

• Pay for knowledge/skills 

• Salary bonuses 

• Remediation and support 

• Probation, reassignment 
or dismissal 


• Professional development 
opportunities 

• Adequate resources 

• Aligned curriculum 
'•'Access to technical assis- 
tance and school designs 


Students 


• Proficiency on academic 
standards 

• Meeting course or 
graduation requirements 


• State and local assess- 
ments 

• Exit/end-of-course 
exams 

• Transcripts 

• GPA, other indicators 


• Promotion to next grade 
or level 

• Test scores placed on 
permanent transcripts 

• High school graduation 

• Diploma endorsement 

• Special recognition 

• Participation in extracur- 
ricular activities 

• Required remediation 


• Opportunity to learn 

• Qualified teachers 

• High quality curricula 

• Remediation programs 


Parents 


• Parent participation and 
volunteerism 

• Communication between 
home and school 

• Parenting skills and 
student support 


• Indicator measuring 
parent involvement in 
schools 


• Parent recognition 

• School choice 


• Parent engagement and 
involvement in schools 


Community/ 

Business 

Leaders 


• Hiring based on student 
transcript 

• Allow parents time off to 
attend school conferences 

• Support education-related 
events 

• Partner with schools 


• Survey to identify busi- 
nesses that use high 
school transcripts in 
hiring process 

• Indicator measuring 
business involvement In 
schools 


* Legislation to require 
parents release time for 
school events 

* Incentives for hiring 
practices 


• Business engagement 
and interest in education 



Source: Many ideas in this chart are drawn from Accountability: Blueprint for Delaware ^ October 1997. 
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Absolute performance. The percent of students 
reported by the proficiency level they meet or exceed 
— regardless of their starting point, past performance 
or demographics. 

Cross-sectional data. Comparison scores of students 
at a given grade level with different students at that 
same grade the prior year. 

Expected score. The anticipated score for a student, 
school or district based on the correlation of back- 
ground factors with student achievement to derive a 
score. This derived score then can be compared with 
actual student, school or district performance. 

Longitudinal data. Data collected from the same stu- 
dents (or a cohort) over time. Longitudinal data can 
take into account statistical determinants of achieve- 
ment such as socioeconomic status and family back- 
ground and are suited to determining the ''value 
added'' over time. 
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can be compared with the percent of proficient 8th- 
grade students from the same school the prior year. 
Although cross-sectional comparisons are less 
expensive to assess, student mobility and differences in 
the student cohorts from year to year can make the 
results misleading.^ 

“Gain scores” reward schools that make progress, even 
low-performing schools, by recognizing and encourag- 
ing continued growth and improvement. Using gain 
scores instead of absolute performance gives every 
school a chance to earn a reward. 

Another way to use gain scores is to create an “expected 
score” based on prior performance, taking into account 
the impact of socioeconomic or other background fac- 
tors. For example, anticipated scores of this year’s 4th 
graders, calculated from last year’s 3rd-grade test, are 
compared with actual scores. This method is a statisti- 
cal compromise of using absolute and relative stan- 
dards, in that each group of students serves as its own 
comparison group. 

• Comparing students using percentile rankings on 
standardized tests. Comparing students using per- 
centile ranks allows districts to use different norm- 
referenced tests for accountability purposes. In 
Arkansas, for example, schools are allowed to 
choose a state-approved norm-referenced test. If 
40% of students are at or below the 25th percentile 
on the approved test, the school is placed in the first 
phase of sanctions. One problem of schools or dis- 
tricts using different tests is that it is nearly impossi- 
ble to aggregate data upward to provide a state-level 
snapshot of student progress toward standards. 

Pros and Cons of Comparison Methods 

Using absolute comparisons gives preference to histor- 
ically advantaged, high-achieving schools. Other 
schools that serve students in low socioeconomic com- 
munities may make large gains in student achievement, 
yet remain below performance standards. But, some 
educators and policymakers believe schools should not 
be rewarded or penalized for factors they cannot con- 
trol. They believe it is fairer either to compare improve- 
ment against schools or school districts with similar 
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socioeconomic and demographic characteristics, or to 
adjust student or school scores to account for the types 
of students served. 

Controlling for student background or prior achieve- 
ment, however, can institutionalize low expectations for 
poor, minority and/or low-achieving students. In effect, 
this practice holds schools with large proportions of 
such students to a lower standard of performance. A 
compromise solution is to use both approaches, reward- 
ing schools for a high level of absolute performance, as 
well as rewarding schools making significant progress. 
This approach would focus attention and effort on com- 
petition among all schools, while still permitting 
schools to strive for rewards relative to others in a 
similar situation. 

Setting Performance Standards, Cut Scores and 
Targets for Improvement 

Performance standards usually are set by panels 
appointed at the state or district level (depending on 
where the standards are set). These panels review stu- 
dent work reflecting a range of abilities and determine 
the level at which that work should be classified. 
Panels typically make judgments individually, discuss 
their rationales and then, aided by statistical programs 
that convert their judgments into proposed scores, con- 
sider the implications in light of actual performance 
data. Then a series of “cut scores” is established, allow- 
ing student performance on the assessments to be con- 
verted into proficiency levels. These levels, such as 
“proficient,” “advanced” and “partially proficient,” are 
used for reporting purposes to identify the percentage 
of students in each category. 

States or districts may want to set targets for improved 
performance. Kentucky has one approach to setting 
performance targets for schools. A score is assigned to 
each level of proficiency on standards. The state’s four 
categories of performance — “distinguished,” “profi- 
cient,” “apprentice” and “novice” — carry scores of 
140, 100, 40 and 0, respectively. Kentucky’s goal is to 
help each school average a score of 100 — possible if 
all students performed at the proficient level or, for 
example, if 50% were proficient, 30% were 



A compromise solution is to use 
both approaches, rewarding schools 
for a high level of absolute perfor- 
mance, as well as rewarding schools 
making significant progress. 
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distinguished and 20% were apprentice (.5x100 + 
.3x140 + .2x40 = 100). 

Under this formula, schools can achieve progress by 
increasing the percentage of “distinguished” students or 
by reducing the share of “novices.” So, the performance 
target for each school in Kentucky is determined by the 
gap between the school’s score and the state goal of 
100. Kentucky wants all schools to reach the goal 
within 20 years, so a school is expected to reduce the 
gap between where it is and where it is expected to be 
by l/20th each year. Most states and districts set annual 
targets for gains between 2% and 5%, but the number of 
years allowed to close the gap varies considerably 
among states. 

POLICY ISSUE 5: 

Deddiog at What Levels To Collect 
and Report Data 

Accountability results can be collected and reported to 
allow comparisons at the international, national, state, 
district, school or individual student level. The levels of 
reporting usually are determined by the purposes of the 
accountability system, as well as by cost considerations. 

• International or National Recent state-by-state com- 
parisons conducted by the National Assessment of 
Educational Progress (NAEP) have allowed states to 
compare student achievement in mathematics and 
reading with other states based on NAEP-developed 
content objectives. Similarly, results from the Third 
International Mathematics and Science Study 
(TIMSS) compare U.S. student achievement with 
results from other countries. States wanting to make 
their own comparisons to other countries in mathe- 
matics and science could have the TIMSS test admin- 
istered with a state sample. Another option is for 
states with similar standards to establish consortia to 
develop some common accountability measures 
allowing cross-state comparisons. 

• State. In the 1980s, a number of states reported stu- 
dent achievement only at the state level. The intent of 
this kind of system was to provide low-stakes moni- 
toring information on how well the education system 



as a whole was doing. Patterned after NAEP, it was 
an inexpensive way to provide monitoring informa- 
tion because of the relatively small number of stu- 
dents that had to be tested. Many of these states, 
however, eventually added data collection at the dis- 
trict and school levels so they could provide reports 
for parents, teachers, administrators and students. 

• District. Collecting and reporting accountability data 
at the district level can identify district-level progress 
toward standards and other established district or 
state goals. States such as Arkansas and South 
Carolina use the district as the unit of accountability 
and provide rewards and sanctions based on district- 
level results. One advantage of using the district as 
the unit of accountability is that it can reduce costs by 
keeping data collection relatively limited (more stu- 
dents have to be assessed on more variables for 
school-level reporting). Another advantage is that 
this approach provides districts flexibility to choose 
assessments aligned to their standards and curricu- 
lum. The disadvantage of using districts as the report- 
ing unit is that the performance of schools within a 
district differs, and few districts have put into place 
accountability structures or can describe school- 
improvement targets for a given year. 

• School. Most states use the school as the reporting 
unit for their accountability systems. In any district, 
some schools will be performing better than others. 
Reporting only district-level results masks these dif- 
ferences. School-level reporting holds each school 
accountable for results, encouraging school staffs and 
their communities to work together to improve stu- 
dent learning. It also allows the district or state to tar- 
get remedial assistance, as well as rewards and 
sanctions. 

• Classroom. Most district reporting mechanisms 
include the classroom as one level of reporting. 
Classroom data can provide information on the rela- 
tive effectiveness of a teacher over time and of the 
curriculum or program used. 

• Student. One reason for testing all students at one or 
more grade levels is to certify students for promotion 
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or graduation. In most states, this data-collection 
strategy is used only at certain grades, although fre- 
quently all students in that grade are tested. About 
half of the states use individual student data to pro- 
mote students from grade to grade or level to level, 
to award a high school diploma or to endorse a high 
school diploma. The remaining states assess students 
primarily as a means of guiding individual student 
remediation or reviewing and improving the school’s 
instructional program. 

A coordinated accountability system can ensure that 
information for all levels of the system is available for 
use by policymakers, educators and community 
members. 

POLICY ISSUE 6: 

Weiglhiiiig tlie Costs, 

A number of design elements affect the costs of 
accountability systems and the student assessments on 
which they rely. Attention should be given to the poten- 
tial costs "of various design elements,’ including the 
following: 

• Custom-developed tests vs/' off-the-shelf” or com- 
mercially available tests. A custom-made test that 
matches a state’s standards is more expensive to 
develop than an ‘off-the-shelf’ norm-referenced test. 

• Selected-response-only v^'. "mixed” assessments. 
Assessments that require a mix of short-answer or 
extended written responses with selected-response 
(e.g., multiple choice) exercises can cost two or three 
times as much as machine-scored tests to score. They 
also may take more time to administer than selected- 
response-only assessments. 

• Number of items assessed. The more items tested, the 
greater the administration and scoring costs. Some 
states deal with this issue by “matrix sampling” test 
items. In a matrix sample, each student takes a por- 
tion of the item set, and statistical techniques are 
used to create school or district scores. 



A coordinated accountability system 
can ensure that information for all 
levels of the system is available for 
use by policymakers, educators and 
community members. 
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• Number of grades assessed. The greater the number 
of grade levels tested, the higher the cost of adminis- 
tration, scoring and reporting. One way to reduce 
costs is to test fewer grade levels, particularly if more 
than one type of test is used. For example, a stan- 
dards-referenced assessment could be used at grades 
4, 7 and 10 in mathematics and reading, while a com- 
parable measure in science and social studies could 
be administered at grades 5, 8 and 11. This approach 
is less costly than assessing students in all grades 
from 4-11. 

• High stakes associated with test results. High-stakes 
tests must be of higher technical quality in order to be 
legally defensible. In addition, for security reasons, 
new forms of the assessment will need to be created 
annually, since the stakes are too high to 
reuse past tests or test items. 

• Extensiveness of the set of indicators to be collected. 
While there may be a desire to be as complete as pos- 
sible in selecting the indicators to be reported along 
with student achievement data, the larger the set to be 
collected and reported, the greater the cost. Some 
elements may already be collected by the state or by 
districts, so the costs of including these data elements 
may be minimal. Other desirable data elements may 
not be currently available and will require additional 
resources to collect, analyze and report. Some data 
elements (for example, the employment of students 
after high school) may be deemed virtually impossi- 
ble to collect on a statewide basis. 

• Size of rewards provided to schools or districts. 
Indiana most recently awarded over $3 million to 
schools; South Carolina awards about $5 million. In 
1995, Kentucky had $26 million available for teacher 
bonuses. The resources associated with providing 
rewards should be considered part of the costs of an 
accountability system. 

The costs of collecting and reporting on each desired 
data element will need to be weighed, including the 
staff time and effort needed to obtain information frorn 
schools and districts. Either the direct or indirect costs 
associated with data collection may limit the extent to 
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which various data elements are included in the 
accountability system. In general, more expensive 
approaches will produce sounder accountability sys- 
tems. But, with finite resources for education, the trade- 
offs between spending resources on an accountability 
system versus putting the resources into classroom 
instruction must be weighed. 

POLICY ISSUE h 
Creating Rewards, Sanctions 
and Other Incentives 

Rewards and sanctions offer the potential for motivat- 
ing school improvement efforts and improving teaching 
and learning (see the ECS publication, A Policymaker's 
Guide to Incentives for Students, Teachers and Schools, 
for more information about incentives). Progress 
toward achieving standards, along with other indicators, 
can be the basis for incentives for high performance and 
disincentives for poor performance. And, including 
other incentives in the system, they also can serve to 
improve motivation and performance.'^ 

Six states — Georgia, Indiana, Kentucky, North 
Carolina, South Carolina and Texas — have created 
rewards and sanctions to encourage schools and dis- 
tricts to increase student performance on standards. 
How progress is measured and reported has significant 
implications for how rewards are allocated and sanc- 
tions imposed, and how the public views the fairness of 
the accountability system. Among the questions policy- 
makers need to address are the following: 

© Will there be absolute standards that must be met 
regardless of the school’s baseline level of perfor- 
mance and factors specific to that school? 

© How should low-scoring but improving schools be 
rewarded? 

© Should consistently high-scoring schools be 
rewarded for their high level of performance in the 
same way as low-scoring but improving schools? 

® What assistance will the state provide to help low- 
performing schools improve? 
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o Who will impose sanctions when schools fail to 
improve? What is the scope of the sanctions? Will 
there be a due-process procedure? What will be the 
cost of imposing the sanctions? 

o What will the rewards be and who will receive them? 
Will the rewards encourage teamwork or will they 
cause dissension and low morale? 



If a state chooses to attach rewards or sanctions to 
accountability results, the accountability system is con- 
sidered “high stakes” (see the ECS publication, 
Education Accountability Systems in 50 States for more 
information about state accountability systems, includ- 
ing the use of incentives and sanctions). If conse- 
quences are to be attached to the results, the 
accountability system needs to be technically and 
legally defensible: it must measure what it says it mea- 
sures and must do so accurately and with multiple mea- 
sures. This step is especially important when 
accountability results are used to make decisions about 
individual students, such as promotion, graduation or 
an endorsed diploma. 

Rewards 

Rewards can come in the form of cash, recognition or 
increased flexibility. They may include cash awards for 
school improvement or to be distributed as staff 
bonuses, freedom from regulations, or statewide recog- 
nition and status. States that provide cash rewards may 
limit how schools spend the money. Some states allow 
the reward money to be used for staff bonuses, although 
Kentucky, for one, is reconsidering this approach.'^ 
Other states ban the use of reward money for staff 
bonuses and athletics. Still others require reward 
bonuses to be used only for school improvement 
efforts. 



How progress is measured and 
reported has significant implications 
for how rewards are allocated and 
sanctions imposed^ and how the 
public views the fairness of the 
accountability system. 



Nonmonetary awards can be given in the form of relief 
from regulations and thus increased flexibility. In 
Texas, a school rated “exemplary” is exempt from 
requirements and prohibitions specified in the state 
school code. The school is still held accountable, how- 
ever, for a lengthy list of code sections, including cur- 
riculum and graduation requirements, elementary 
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class-size limits and programs for students who are dis- 
abled, bilingual or at riskJ^ 

South Carolina grants broad exemptions to qualifying 
schools. A school must have received an incentive 
award twice over a three-year period in addition to other 
requirements. Flexibility status earns release from 
requirements of the defined minimum program. 
Qualifying schools also are freed from regulations gov- 
erning class scheduling, class structure and staffing. ’’ 
Exempt schools must continue to participate in the state 
assessment program and score well to retain deregu- 
lated status. 

In some states, such as Indiana, monetary awards are 
supplemented with noncash awards or public recogni- 
tion. The use of such awards (called “Four Star” schools 
in Indiana) originated with the desire to recognize high- 
performing schools that did not earn awards based on 
growth scores. Four Star schools are those that perform 
in the top quartile of schools in each of four indicators 
and satisfy all expected performance levels defined in 
Indiana’s school accreditation program. The Four Star 
designation recognizes high-capacity districts, while at 
the same time allowing a larger amount of money to go 
to poorer school districts that improve. 

Sanctions 

Sanctions usually are implemented on a continuum, 
starting with assistance and possibly ending with clos- 
ing a school or the state taking it over if adequate 
progress is not being made. Twenty states have enacted 
“academic bankruptcy” laws that allow state interven- 
tion in severely troubled districts.^* Early responses to 
academic bankruptcy can include such things as public 
hearings or the assignment of onsite technical assis- 
tance, as is available under Kentucky’s Distinguished 
Educator Program. If progress is not made, the possibil- 
ity exists of a state takeover of the district and other 
forms of intervention. Other options include changing 
management of the district (school board or school 
superintendent) or school (principal and teachers), 
allowing parents to choose another school or providing 
additional resources. Most states have accountability 
mechanisms focused on identifying low-performing 






schools. Some states, however, also impose sanctions 
on districts with a large number of low-performing 
schools. 

States are exercising considerable caution as they 
implement sanction programs. The problem with high- 
stakes accountability systems is they must be backed up 
with technically accurate and legally defensible assess- 
ments and procedures. To be conservative, many states, 
such as Maryland, are identifying only a handful of 
schools generally agreed to be at the lowest level of stu- 
dent performance. 

Florida’s accountability program was developed over 
several years before 1996 legislative action gave the 
state superintendent authority to intervene in the opera- 
tion of districts that failed to make adequate progress.’^ 
Delaware may allow the use of an alternative assess- 
ment and other indicators instead of the state assess- 
ment results to report student results using proficiency 
levels. 

Do Rewards and Sanctions Work? 

With the exception of South Carolina, state reward and 
sanction programs have been in effect for only a few 
years. Early evidence suggests, however, that sanctions 
do motivate schools to do better. But the results are 
mixed regarding the impact of rewards. 

Richard A. King and Judith K. Mathers,^® in an analysis 
of four states with high-stakes accountability systems, 
made the following conclusions regarding the effect of 
rewards on teachers’ motivation: 

• Extrinsic rewards may contribute to reform imple- 
mentation but may not motivate change. 

• Extrinsic factors pale in comparison with the intrin- 
sic aspects of teachers’ work. 

• State and local attention to improvement efforts has 
as great an incentive effect as the money associated 
with such status. 

Regarding the effect of rewards on schools and class- 
rooms, these same researchers concluded the following: 
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• Changes in classroom practice are present in both 
reward and nonreward classrooms in the four states. 

® Leadership and other factors may be more critical 
than monetary rewards in change efforts. 

• Rewards and sanctions may stimulate conversations 
with professionals and parents on school improve- 
ment. 

• More attention is given to racial/ethnic minority 
group students in states in which rewards and sanc- 
tions take such students’ performance into account. 

The impact of rewards on improving student test scores 
is more difficult to ascertain. Few studies are available 
that include findings on the effects of rewards on stu- 
dent achievement. 

Some state leaders believe the mere threat of sanctions 
is a more powerful motivator than the distribution of 
financial rewards. Local school leaders want to avoid 
the negative publicity associated with low performance 
or the stigma attached to being designated a “school in 
decline.” But sanctions do seem to work. Kentucky 
reports that most of the schools targeted as being “in 
decline” in 1994 appear to be making enough progress 
to ensure that more extreme sanctions are not imposed.^* 
In more extreme cases, however, such as state takeovers 
or the reconstitution of districts, the long-term effects of 
sanctions are less clear. 

Cautions for High-Stakes Testing 

High-stakes accountability systems can produce some 
unintended, negative consequences. These can include 
the narrowing of instruction, de-emphasis on untested 
subject areas, test-score inflation through inappropriate 
or questionable test preparation practices, and staff 
morale problems. 

Research shows that in a high-stakes system^^ what gets 
tested gets taught, and that the converse is also true — 
what is not tested is not taught. High-stakes tests that 
assess student performance on only one or two content 
areas — reading and mathematics, for example — will 
have an adverse affect on how much science or social 




studies is taught. Similarly, accountability systems that 
consistently assess only some standards within a con- 
tent area likely will result in limited instruction on the 
standards not assessed. 

Inflation of test scores was first noted in a 1987 study 
that showed that many states had an implausible pro- 
portion of districts reporting themselves to be “above 
the national average.”^"^ Test-score inflation can be due 
to many factors, including making comparisons using 
outdated norms, and teaching to the test. 

Staff morale problems can occur when rewards are 
selectively awarded, such as one state’s recent policy of 
providing cash rewards only to principals of high- 
performing schools. 

POLICY ISSUE 8: 

Melpieg the Pelblk Uedeirstainid Resmlts 

How the public, policymakers and educators view — 
and whether they understand — the accountability sys- 
tem'has'a lorto"do with its success. One' challenge is 
communicating clearly the results of the accountability 
system to policymakers — both in terms of the report 
cards and the system evaluation. A second challenge is 
to ensure that parents, teachers and the public see the 
accountability system as fair. To achieve this, these 
groups need to understand how the accountability sys- 
tem works, what led to specific rewards and sanctions 
being used, and why such actions are likely to be 
productive. 

State leaders report that parents are confused about the 
difference between the new tests that report student 
progress toward standards and the familiar norm-refer- 
enced tests in which a single score gives a percentile 
rank against a national norm.^^ Standards-based 
accountability systems place students in categories such 
as “partially proficient,” “proficient” or “advanced” 
based on their assessment scores. Parents may not 
know what it means to be “partially proficient” and may 
be more comfortable with percentile-rank comparisons 
than with assessment of learning progress over time. 
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COMMUNICATING NEW ASSESSMENT 
RESULTS TO PARENTS 



To assist educators in explaining the new state assess- 
ment results to parents and the public, CONNECT, 
Colorado Statewide Systemic Initiative, and a busi- 
ness coalition, Teaming for Results, created a 
teacher's guide to the Colorado Student Assessment 
Program. 

To explain the difference between criterion-referenced 
tests and norm-referenced tests, Colorado uses the fol- 
lowing example: Picture a group of students climbing 
a mountain. A norm-referenced test tells whether a 
given student is in the lead, near the middle of the 
group or lagging behind. It will not show where on 
the mountainside (near the top, middle or bottom) 
each student is — only their relative position to one 
another. The state assessment, on the other hand, is 
based on the state model content standards (a crite- 
rion-referenced test) and provides student achieve- 
ment information based on proficiency levels. 
Proficiency levels show how well a student is meeting 
the state standards, or, in this case, where the student 
is located on the mountain. 

Source: How Are Colorado Students and Schools 
Measuring Up?: A Teacher’s Guide to the Colorado 
Student Assessment Program. CONNECT and 
Teaming for Results, 1997. 
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Issues such as low scores, the consequences of test 
results and costs sometimes arise and will need to be 
explained.^® 

Many of the “lessons learned” about involving the pub- 
lic in establishing standards and developing new assess- 
ments also apply to the design and implementation of 
standards-based accountability systems. They include 
the following: 

• Public involvement is inseparable from policy devel- 
opment and implementation. 

• Listen to people first, talk later. 

• If you do not involve teachers, expect to fail. 

• Show how new approaches enhance rather than 
replace old methods. 

• Expect criticism and respond to it, but do not become 
preoccupied with it. 

• Understand that it takes time to get results, and set 
expectations accordingly. 

• Demystify reforms. Do not just tell people about new 
standards, assessments and accountability processes; 
show them examples. 

• Communicate the big picture to the public while 
reserving technical discussions for teachers and oth- 
ers directly involved in implementation. 

• Be willing to adjust plans based on the needs and 
concerns of educators, parents and the public. 

• Build a strategic communications plan designed 
around people’s questions and needs, and use it 
throughout the design and implementation process 
(see the ECS publications. Building Community 
Support for Schools: A Practical Guide to Strategic 
Communications, Do-It-Yourself Focus Groups: A 
Low-Cost Way To Listen to Your Community, So You 
Have Standards . . . Now What? and Listen, Discuss 
and Act: Parents' and Teachers' Views on Education 
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Reform for more information on communications 
plans and efforts). 

POLICY ISSUE 9: 

Siipportieg Teadhers, Schools 
aed Bnstricts 

States can play several different roles in providing sup- 
port for educators, including creating standards for 
licensure or certification and professional development, 
organizing professional development centers, providing 
additional resources, developing reform networks, or 
providing direct assistance to schools or districts. 

Ensuring Professional Accountability 

State efforts to ensure accountability need to focus as 
much on creating the conditions for effective teaching 
as they do on holding schools accountable. Raising aca- 
demic expectations for students will work only if teach- 
ers have the knowledge and skills to make good 
decisions and teach in ways that help all students meet 
higher levels of achievement. Increasing accountability 
should be coupled with investments in improved 
teaching. 

The recent report of the National Commission on 
Teaching and America’s Future^^ noted the failures of 
recent reforms are due not to schools’ unwillingness to 
improve, but to the facts that most educators do not 
know how to implement the reforms, and their work 
environments are not structured to help them do so. If 
increased student achievement is the goal, it will need 
to be coupled with accountability policies that ensure 
teachers and other educators have the knowledge and 
skills they need to teach effectively to the new stan- 
dards and help schools evaluate and reshape their 
practices.^^ 

To ensure teachers have the knowledge and skills they 
need to teach to the new standards, many states are 
putting in place teacher licensure or certification stan- 
dards. These can include what constitutes effective 
teaching and what teachers should know and be able to 
do (such as knowing core content, adapting teaching for 
diverse learners, using multiple assessment strategies, 
using a variety of instructional strategies, etc.). 




State efforts to ensure accountability 
need to focus as much on creating 
the conditions for effective teaching 
as they do on holding schools 
accountable. 
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The impact of professional teaching standards can be 
extended by state support of ongoing teacher profes- 
sional development. This could include licensing teach- 
ers based on performance tests of subject-matter 
knowledge, teaching knowledge and teaching skill tied 
to standards developed by the Interstate New Teacher 
Assessment and Support Consortium. Another option is 
to create mentoring programs for beginning teachers 
that include assessment of teaching skills as the basis 
for a continuing license. Standards and assessments of 
the National Board for Professional Teaching Standards 
could be the standard for accomplished teaching and the 
basis for rewarding high levels of knowledge and skill. 

Supporting Professional Development 

States play a number of roles in supporting professional 
development.^^ In addition to taking the lead in devel- 
oping professional standards for teacher preparation, 
certification or licensure, and professional develop- 
ment, states are providing assistance and resources for 
professional development opportunities. States such as 
Delaware and Ohio have developed regional profes- 
sional development centers providing training and sup- 
port for teachers. Other states, such as Utah, appropriate 
2% of the Minimal School Program for state develop- 
ment activities at the local level and $1 million for 
statewide preparation activities. 

Utah encourages districts to create a staff development 
policy, designate a staff development coordinator, align 
staff development activities with improvement goals 
and evaluate the programs. The Utah Department of 
Education also publishes an annual listing of summer 
inservice activities conducted by the state and other 
providers such as higher education institutions and local 
districts. Vermont involves teachers in the scoring of its 
state assessment program as yet another approach to 
professional development. Regional networks of teach- 
ers are organized to provide training on portfolio scor- 
ing and instructional training to help students create 
portfolios. Many states also support mentor teacher pro- 
grams ranging from providing competitive grants to dis- 
tricts to create mentor programs to developing 
academies that train experienced teachers and adminis- 
trators to become in-school consultants. 



States also are providing professional development to 
administrators, usually through administrator acade- 
mies. To lead instruction that supports students achiev- 
ing high standards, administrators also must understand 
how to help teachers with instructional approaches, how 
to use data for student learning and school improve- 
ment, and how to share decisionmaking with teachers 
and school community members. 

Promoting Local Accountability 

School-level accountability can include a number of 
different strategies. First, schools need student achieve- 
ment and other information to evaluate their effective- 
ness. District accountability systems should collect data 
on important goals and in grades and subject areas 
where the state accountability system does not collect 
information. 

States can support the collection of additional data 
through their state assessment programs by requiring 
and supporting the administration and scoring of local 
assessment data, with a sample of the results reported to 
the state and included in the state accountability data. 
For example. North Carolina plans to place numerous 
performance tasks and scoring guides on a World Wide 
Web site which all teachers can access. Teams of edu- 
cators from each district will score student portfolios 
each fall; the results will inform classroom instruction 
and be part of a local accountability system. 

Schools may want to implement their own accountabil- 
ity systems such as requiring a student exhibition for 
graduation and setting up school-quality review com- 
mittees. Mastery exhibitions require students to demon- 
strate their knowledge in front of review committees. 
This public demonstration of knowledge allows teach- 
ers and the community to talk about the meaning of 
achieving standards and the quality of student work. 
Constructive debates can help teachers interpret school 
goals and develop effective strategies for achieving 
them, which can lead to staff seeking new knowledge 
and skills. 

School-quality reviews can be another strategy to foster 
local accountability. Teams of practitioners, possibly 
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including state department of education staff, spend a 
week in individual schools to examine teaching and 
learning practices. Feedback is provided to foster 
improvement, and the school-quality reviews also can 
result in descriptive evidence of how well the school is 
doing. This evidence can become a baseline for improv- 
ing teaching and learning practices over the next several 
years. 

States can support local accountability by creating 
expectations that individual schools establish their own 
standards for performance and reporting systems. It can 
support staff-development opportunities to help teach- 
ers within a school formulate performance goals and 
ways to implement them. Districts and states can estab- 
lish and support reform networks to assist in sharing 
best practices and procedures for evaluating student 
learning and school goals and, most important, acting 
on the results. 

Professional Development Centers 

A number of states have taken a more active role in sup- 
porting or providing professional development. In addi- 
tion to rethinking certification or licensure standards for 
teachers, several states are creating professional devel- 
opment centers."^® 

Ohio has 12 regional professional development centers 
for teacher training and development. These centers 
create and coordinate professional development oppor- 
tunities and assist district staff with school improve- 
ment strategies. These centers also serve as a regional 
structure for many state programs such as school-to- 
work and tech prep. The centers build collaboration 
among school districts, institutions of higher education 
and private providers, and link resources and expertise 
to schools and districts. 

Delaware also has a state-supported teacher center 
which grew out of collaborative efforts among several 
school districts. The center now manages eight regional 
professional development centers which provide inser- 
vice workshops, free instructional materials, on-loan 
equipment and a place and opportunity for teachers to 
share ideas about best practices. 







Providing Additional Resources 

Several states provide additional resources for schools 
or students who are achieving below grade level. For 
example, Alabama provides funds to schools for tutor- 
ial assistance for students one grade level or more 
behind the norm. Indiana provides grants to help 
schools provide remediation for students who score 
below state proficiency standards and prevent future 
remediation of students at risk of falling below the stan- 
dards. 

In 1996, Delaware authorized $200,000 for student 
tutoring and mentoring activities, distributed competi- 
tively to K-5 schools to establish programs for students 
academically at risk. Programs must address one-on- 
one tutoring, early-childhood preventive intervention 
strategies, parental involvement, and program and stu- 
dent performance evaluation. 

Mississippi has established’ a Support Our Students pro- 
gram within the Department of Education to award 
grants to community and neighborhood organizations 
that provide high-quality after-school programs for chil- 
dren in grades K-9 and comprehensive delivery of ser- 
vices to those children. 

New Jersey runs nontraditional high schools in each of 
its 21 districts, most on college campuses. At-risk stu- 
dents get extra support services and one-on-one coun- 
seling to help with academic and behavior problems. 
Each student’s program is tailored to his or her individ- 
ual needs and may even include enrolling in college 
courses. 

Ohio has taken an innovative approach to supporting 
school improvement. The Venture Capital program pro- 
vides “risk capital” of up to $25,000 for each of five 
years to help schools improve. Venture Capital grants 
are designed to be long-term, evolving efforts focused 
on a particular dimension of change, e.g., curriculum 
development, professional development, assessment. 
After the five-year time frame, schools are expected to 
have made significant progress in institutionalizing 
their commitment to professional development and 
transforming the culture in which school renewal is 
implemented. 
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Reform networks can help educa- 
tors, schools and districts learn 
about approaches to instruction, 
curriculum design, assessment, 
professional development and 
management strategies that have 
proved effective in other settings, 
and assist them in adapting these 
approaches to their own 
circumstances. 



Reform Networks 

Reform networks are interconnected groups of educa- 
tors, schools or districts with a common interest in a 
specific reform approach.^' Some networks focus on a 
specific subject, such as reading or mathematics, while 
others target specific topics, such as assessment, liter- 
acy or minority student achievement. Still other net- 
works offer a focused approach to reform and 
restructuring through the use of a specific “whole- 
school” design that links all aspects of the school or dis- 
trict — curriculum, instruction, assessment, staff 
training and school management. Other networks rely 
on more formal structures and processes, including a 
central staff that provides teachers and school leaders 
with the information and expertise they need to 
improve their knowledge and skills in specific ways. 

Reform networks can help educators, schools and dis- 
tricts learn about approaches to instruction, curriculum 
design, assessment, professional development and man- 
agement strategies that have proved effective in other 
settings, and assist them in adapting these approaches to 
their own circumstances. Networks provide a wide 
range of opportunities for professional growth and 
enrichment by engaging members in varied activities 
— curriculum workshops, leadership institutes, intern- 
ships, conferences, study groups, electronic bulletin 
boards — designed around participants’ needs and 
interests. 

Some states that have enacted academic bankruptcy 
laws have found networks to be a useful tool for 
improving performance of chronically low-achieving or 
mismanaged schools. In several states, including 
California, Maryland and New Jersey, a school identi- 
fied as failing may be required to affiliate with a reform 
network as part of its tum-around plan. States also may 
take an active role in serving as a clearinghouse or bro- 
ker for networks. 

Providing Direct Assistance 

Kentucky’s Distinguished Educator program is perhaps 
the best known example of a state-directed program 
providing assistance to low-performing schools. 
Created as part of the Kentucky Education Reform Act, 
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Distinguished Educators are selected from a pool of 
applicants to serve as teaching ambassadors to schools 
designated “in decline” or those that request the extra 
assistance. At the same time, Kentucky adopted the 
School Transformation Assistance and Renewal 
(STAR) project for assisting schools “in decline.” 
STAR provides resources to schools and defines the 
change process Distinguished Educators use to help 
improve low-performing schools. The majority of 
schools with Distinguished Educators have improved 
student achievement such that they were no longer des- 
ignated as “in decline.” 

POLICY ISSUE Wi 
Fiee-TRimiinig tlhe System 

The success or failure of an accountability system is 
determined not only by its design but also by the level 
of commitment policymakers demonstrate to its ongo- 
ing implementation and refinement. Without such sus- 
tained commitment, an accountability system may be 
perceived as just another program imposed by the state. 

It is important, for example, that proposed legislation 
be carefully monitored to ensure consistency with the 
accountability design. The state accountability panel or 
task force should be alert to initiatives that not only 
would be inconsistent and detrimental, but also would 
send a message to the education community that com- 
mitment to the accountability system is waning. 
Similarly, state education department and other agen- 
cies’ policies must be reviewed for potential obstacles 
to implementing the accountability system. 

Other Key Issues To Consider 

• Standards and assessments should be updated and 
refined. As skill needs change, content standards 
should be adjusted to reflect these, as well as the 
assessments that measure them. 

• The state's budget must be aligned to the account- 
ability design, not only within the education portion 
of the budget, but also in other areas such as human 
services and health (for example, pre-K programs, 
juvenile justice and safe-school programs). 




The success or failure of an 
accountability system is determined 
not only by its design but also by the 
level of commitment policymakers 
demonstrate to its ongoing 
implementation and refinement. 
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• The state education department's relationship with 
the accountability panel, as well as its responsibili- 
ties for implementation, must be clearly defined. The 
department’s implementation plan must have a deliv- 
ery system capable of providing the necessary tech- 
nical assistance. It also must have the ability to 
disseminate information in a consistent, clear and 
effective manner. 

• Training should be a high priority. Beyond the initial 
training in how the accountability system will work, 
there will be a need for ongoing training of teachers, 
principals, superintendents, school board members, 
university professors and school advisory council 
members. 

• Attention must be paid to the role of postsecondary 
institutions. Not only will they be the recipients of 
students held to the state’s standards, they also will 
produce the teachers who must have the skills to 
teach to the standards. 

• The business community needs to be involved. 
Business community involvement may be in the form 
of technical expertise at the school or district level; 
influence in shaping state policy; and active, ongoing 
support of the state’s standards. 

• Newspaper editorial boards should be kept up-to- 
date on the development and implementation of the 



accountability system. If they are not kept within the 
loop, visibility of the accountability efforts will be 
lessened, and the potential for misinformation will be 
increased. 

• The state accountability panel should have responsi- 
bility for ongoing evaluation and refinement of the 
accountability system, including the following: 

— Conducting an annual survey of schools and dis- 
tricts (this could be a random sample) to provide 
feedback on the status of implementation, the 
extent and effectiveness of the supporting infra- 
structure, and the identification of obstacles to 
implementation 

— Preparing an annual report to the legislative edu- 
cation committees and state board of education 
on the status of implementation 

— Annually reviewing the design and structure of 
the accountability system and making recom- 
mendations to the legislature on statutory 
changes, to the state board on rule changes and to 
the state department of education on administra- 
tive policy changes 

— Conducting a longitudinal study of the impact of 
the accountability system on student perfor- 
mance. 
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CONCLUSION 



“A clear picture of what you want to accomplish, a 
comprehensive measurement system to gauge progress 
and a commitment to act on the results to make appro- 
priate changes” — those were Colorado Governor Roy 
Romer’s requirements for improving student perfor- 
mance. Content and performance standards are 
intended to provide the “clear picture” of what needs to 
be accomplished. A standards-based accountability sys- 
tem is the comprehensive measurement system to 
gauge progress. Demonstrating the “commitment to 
act,” by supporting schools and districts with profes- 
sional development opportunities, by engaging all stu- 
dents, and by securing broad public support and 
involvement — that is the challenge that remains. 



ERJC 
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APPENDIX A 



FREQUENTLY USED INDICATORS IN STATE ACCOUNTABILITY SYSTEMS 



Level 



Purposes of Accountability 
System 



Primary Design Features of 
an Accountability System 
Based on Purposes 



Accomplishing the Purposes; 
Uses of Accountability 
Information 



State 



' Monitoring system progress 

' Identifying the relative perfor- 
mance of one system to 
another 

' Holding districts and schools 
accountable 

Providing targets for 
instruction 

' Providing information for pro- 
gram improvement 

' Allocating resources 
’ Certifying students 



• Indicators could include 
information on expenditures, 
district or school demographics, 
program quality. 

• Assessment is aligned to state 
standards and assesses only 
core academic standards. 

* At least three grades are 
assessed — one in elementary 
school, one in middle school 
and one in high school. 

* Assessment item formats are 
primarily selected-response with 
some short-answer, extended- 
response and performance 
tasks to assess more complex 
standards. 

► If accountability system is “high 
stakes,” then assessment meets 
high technical and legal criteria. 



• Providing rewards and sanctions 
for high- and low-performing 
districts or schools 

► Revising state policy to improve 
adequacy and equity 

► Reallocating resources to where 
they are most needed 

» Targeting assistance to low-per- 
forming schools 

► Providing information to the 
public on the status of schools 
and the education system 

» Endorsing diplomas or providing 
certificates of mastery 

» Focusing teachers’ attention on 
standards 



District 



• Monitoring district progress 
' Improving student performance 

> Holding schools accountable 

' Providing targets for instruction 
' Allocating resources 

► Evaluating program effectiveness 



► Indicators collect information 
on school demographics, 
school-level expenditures, 
school program, teacher quality 
and parent satisfaction. 

► Assessment is aligned to 
district standards and may be 
administered at every grade or 
at grades not assessed by the 
state accountability system. 

► District assessment results are 
reported to the community and 
may be included in the state 
assessment results. 

» Assessment item formats 
include selected-response and 
can include more short-answer, 
extended-response, perfor- 
mance tasks and portfolios to 
assess more complex skills 
and variations in school designs 
while meeting technical 
accuracy. 



► Providing rewards and sanctions 
for high- and low-performing 
schools (including determining if 
schools meet their performance 
contract agreements) 

► Reallocating resources to 
schools that need additional 
assistance 

► Identifying effective programs 
such as whole-school designs 

► Providing comparisons with 

, other (similar) schools in the 
district 

► Reporting school-level data to 
the public to promote conversa- 
tions related to school designs 
and school improvement: creat- 
ing a consumer guide to district 
schools 
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APPENDIX A (Continued) 



FREQUENTLY USED INDICATORS IN STATE ACCOUNTABILITY SYSTEMS 



Primary Design Features of Accomplishing the Purposes: 
Purposes of Accountability an Accountability System Uses of Accountability 

Level System Based on Purposes Information 



School 



• Providing targets for instruction 

• Evaluating program effectiveness 

• Placing students 

• Certifying students 

• Improving student performance 



• Indicators could include com- 
munity characteristics, student 
demographics, attendance rates, 
graduation/completion rates, 
tracking post-school student 
performance, school-level inputs 
such as resources allocated to 
professional development, 
instructional materials and 
degree of implementation of 
school program. 

• Assessments are aligned with 
school goals and school pro- 
gram. 

• Assessment item formats can 
include selected-response, short- 
answer, extended-response, per- 
formance tasks, portfolios and 
mastery exhibitions. 

• Assessment results are available 
in time to make improvements. 



• Input for school’s strategic plan- 
ning process 

• Evaluating how well schools 
meet their goals 

• Evaluating effectiveness of 
school design for student popula- 
tion and community needs 

• Input for reallocating resources 
within the school 

• Basis for communitywide conver- 
sations on school improvement 

• Gauge parent satisfaction with 
the school program 

• Provide information for district’s 
consumer guide to schools 



Classroom 



• Student diagnosis 

• Improving student performance 



• Assessments are aligned with 
school program and are appro- 
priate for the grade or student 
group. 

• Assessment results are immedi- 
ately available to use for student 
diagnosis to improve student 
performance. 



• Provide information on individual 
students for communication with 
parents 

• Provide information to place or 
select students 

• Provide information on effective- 
ness of classroom teaching and 
learning 
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APPENDIX B: GLOSSARY OF TERMS 



Accountability. Accountability systems provide infor- 
mation that tells policymakers, the public and others 
how well the education system — classrooms, schools 
and districts — is doing. Information typically includes 
student assessment data and indicators such as dropout 
and graduation rates. Accountability information can be 
used in different ways: to provide information to the 
public, to help all the groups involved reach agreement 
on how to improve the system, or to provide rewards or 
sanctions for success or failure. 

Accreditation. Accreditation is the review process a 
school or district undergoes periodically to ensure that 
it meets state requirements and quality measures. The 
process usually involves the review of a district or 
school’s evaluation procedures and improvement plans; 
the effectiveness of education programs and services; 
and other policies, practices and management 
processes. Recently, some states began using accredita- 
tion as a performance-based process focused on student 
achievement and school-improvement planning. 

Assessment. Assessment is the measurement of what a 
student knows and is able to do, usually expressed in 
terms of progress toward a standard or mastery of a 
standard. Assessment can include diverse measures, 
such as multiple-choice tests, constructed-response 
exercises, performance measures and portfolios. 

Content standards. Content standards are statements 
that define what students should know and be able to do 
in various subject areas and at different points in their 
education. 

Criterion-referenced test. This test is designed to pro- 
vide a clear picture of what a student knows and can do. 
It measures performance against an established criterion 
or standard, rather than in comparison to a norm group. 

Cross-sectional data. Cross-sectional data compare 
scores of students at a given grade level with different 
students at that same grade the prior year. 

Expected score. The anticipated score for a student, 
school or district based on the correlation of back- 
ground factors with student achievement to derive a 



score. This derived score then can be compared with 
actual student, school or district performance. 

Indicator system. The system of measures of inputs, 
processes and outcomes used to describe the perfor- 
mance of a school, district or state. 

Longitudinal data. Data collected from the same cohort 
of students over time. Longitudinal data can take into 
account statistical determinants of achievement such as 
socioeconomic status and family background and are 
suited to determining “value added” by a teacher or 
school. 

Norm-referenced test. This test is designed to compare 
a student’s test score with those of students in a norm or 
comparison group, such as a nationally representative 
sample. More generally, it has come to refer to compar- 
isons of performance among students, schools, districts 
or states. 

Performance assessments. Performance assessments, 
also called “authentic” or “direct” assessments, measure 
actual, demonstrated skills. Many times, these are con- 
structed or written responses to questions on an assess- 
ment. 

Performance standards. Performance standards pro- 
vide explicit definitions and concrete examples of how 
well students are expected to learn the material repre- 
sented by content standards. Performance “levels” also 
may be used to define students’ demonstrated profi- 
ciency at various points as they progress toward a stan- 
dard. 

Portfolios. Portfolios are collections of student work 
designed to show progress over time, level of accom- 
plishment and/or the student’s best work. 

Stakes. Stakes are the consequences tied to perfor- 
mance on an assessment or test. A “low-stakes” test has 
few or no consequences tied to results; a “high-stakes” 
test has consequences related to performance. Stakes 
can include rewards for high performance, sanctions for 
low performance, promotion or graduation. The 
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rank-ordering of schools or districts when test results 
are publicly reported can be considered “high stakes.” 

Value added. The amount of student achievement “con- 
tributed” by a teacher or school during a school year or 
other set period of time. A school’s contribution to 



students’ achievement is the “value added” by the 
school. How much schools add to students’ learning can 
be calculated by assessing the differences in fall-to- 
spring (or fall-to-fall or spring-to-spring) testing of the 
same students. 
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ADDITIONAL RESOURCES FROM ECS 



Here are some other related publications from ECS that 
you’ll find useful for getting citizen input and making 
decisions about education reform: 

Accoeetobility 

Education Accountability in 50 States 

How does your state’s accountability system compare 
with other states? This publication defines the compo- 
nents of a performance-based education accountability 
system and then shows what each state’s system looks 
like. Tables illustrate which states have standards and 
assessments, multiple indicators, rewards and sanc- 
tions; whether they are based in statute or regulation; 
how decentralized states compare to centralized states; 
and other information. 1997 (SI-97-12), 23 pp., $7.50 

A Policymaker Guide to Incentives for Students, 
Teachers and Schools 

The goal of dramatically raising student achievement 
cannot be accomplished without changing the incen- 
tives for students, teachers and schools. This guide is 
designed to heighten policymakers’ understanding of 
the choices and challenges related to creating incentives 
for these groups. It identifies the elements for improv- 
ing performance in a standards-based system; describes 
incentives for students, including using standards for 
promotion and admissions to higher education or entry 
into apprenticeship programs; highlights incentives to 
increase teachers’ knowledge and skills; describes 
incentives for schools, including reallocating resources, 
boosting achievement though school awards and imple- 
menting extracurricular programs; and finally, looks at 
implications for state policymakers in creating incen- 
tives. 1997 (AN-97-5), 44 pp., $10.00 

Accountability — State and Community 
Responsibility 

A decade ago, the idea that states could, or would, step 
in and intervene in local school districts because stu- 
dents were not achieving enough was almost unthink- 
able. Now, however, state intervention not only is 
acceptable among a majority of the states, but is becom- 
ing a major policy tool to deal with seriously under- 
achieving schools or districts. Policymakers shared 
“lessons learned” at a recent ECS conference in Ohio; 



this paper summarizes that discussion. 1998 (SI-98-2), 
20 pp., $6.50 

Commeekattioes 

Building Community Support for Schools: A Practical 
Guide to Strategic Communications 
Learn how to build credibility, trust and stronger ties to 
the community by involving the community as change 
is considered — not after the fact. This guide focuses on 
the practical tools you need to develop strategic com- 
munications with education colleagues, parents and the 
general public. A step-by-step plan will help you set 
more thoughtful priorities, spend scarce resources more 
wisely, listen and respond to the needs and concerns of 
people inside and outside the education community, tar- 
get activities to those who are most affected by and con- 
cerned about education improvement efforts, and 
regularly measure results. 1997 (AN-97-3), 32 pp., 
$ 10.00 

Do-It-Yourself Focus Groups: A Low-Cost Way To 
Listen to Your Community 

What is important to people in the school community? 
What changes do they want to see in their schools, and 
how do they want to participate in decisionmaking? 
Learn how to use the valuable information that informal 
conversation often yields. This guide helps you gather 
information in a systematic way, organize what you 
hear, report back what you’ve learned to the community 
and, most important, use that information to guide 
school change. 1997 (AN-97-2), 31 pp., $10.00 

Americans Public Schools Must Change . . . But Can 
They? 

Half the voters in the last national election say their vote 
was affected by a candidate’s stand on public education. 
How important is education in selecting political candi- 
dates? What role do people expect national leaders to 
play in improving education? And what kinds of change 
do people want to see in public schools? These and 
other questions are answered in a detailed report of a 
national survey ECS conducted following the 
November 1996 general election. Includes charts and 
demographic breakdowns. 1997 (AN-97-1), 26 pp. 
$ 10.00 
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Let^s Talk About Education Improvement 

Everyone has an opinion about education and they want 
to talk about it. You can help foster this dialogue and 
involve all segments of your community in discussions 
about how public schools can do a better job of educat- 
ing students. This conversation guide offers tools for 
planning and holding a meeting, discussion activities, 
follow-up and take-home materials, and lists additional 
resources. 1996 (AN-96-3), 27 pp., $10.00 



Listen, Discuss and Act: Parents ^ and Teachers ^ Views 
on Education Reform 

What do the people closest to students think of educa- 
tion improvement efforts? Any communication about 
changes must start by listening to the people most 
affected by the changes. This report details the results of 
parent surveys and parent and teacher focus groups in 
seven cities and states. Includes recommendations for 
building public and political support for education 
improvement. 1996 (AN-96-2), 36 pp., $10.00 



Listeny Discuss and Act: Focus Groups About ~ 
Improving Education 

A powerful video companion to the report Listen, 
Discuss and Act: Parents' and Teachers' Views on 
Education Reform. Hear what parents and teachers say 
about standards, parent involvement, assessment and 
accountability, and what they think gets in the way of 
efforts to improve schools. 1996 (AN-96-V2), 19 min., 
$10.00 





Jargon 



^ Winner of a Telly Award for out- 
^ standing video, this video provides a 
light-hearted look at a serious subject: 
how the language used by educators and educa- 
tion policymakers often obscures meaning and under- 
mines support for reform. 1995 (SM-95-V4), 5: 14 min., 
$8.00 



Edecatioini Reform 

The Progress of Education Reform: 1997 

This year’s report examines research data on student 

achievement, public attitudes and demographic trends 




with a brief interpretation of what the data means. 
Major education reform efforts and state actions under 
way are analyzed as to their effectiveness in improving 
student achievement. 1998 (SI-98-1), 51 pp., $12.50 

A Policymaker^s Guide to Education Reform 
Networks 

Get the jump on the new federal money for educators 
interested in affiliating with a reform network! This 
publication tells you the kinds of networks available, 
how they work, and the benefits and services they offer. 
An easy-to-read-and-compare chart provides a thumb- 
nail sketch of some major education reform network 
models. 1997 (SI-97-11), 40 pp., $10.00 

The State Education System — 

A Continuum of Systemic Change 

What constitutes systemic change? Where are you as a 
state in the change process? This chart describes differ- 
ent stages of systemic change in five key policy areas: 
vision, policymaking and alignment, teaching and 
learning, public and political support, and networks and 
collaboration. An effective dialogue tool for policy- 
makers and lay audiences. 1993 (SI-93-1), wall chart 
(18 X 24 inches), $2.00 

Creating a Flexible, High- 
Performance Education System 
This set of charts provides descriptions of developmen- 
tal stages for an education system that is moving from a 
regulatory system to one that encourages and supports a 
diverse set of schools. This tool describes the state pol- 
icy changes necessary to create the flexibility needed 
for a high-performance system. Five key areas 
addressed include: standards, curriculum, assessment 
and accountability; decentralizing decisionmaking to 
the school level; professional development; resource 
reallocation; and public engagement. This tool can pro- 
vide a basis for dialogue, self-assessment and action 
planning to move the system toward desired goals. 1996 
(SI-96-10), 13 pp., $5.00 
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A Policymaker Guide to Standards-Led Assessment 

Learn how standards-led assessments closely link what 
is taught to what is tested. The helpful guide examines 
the important challenges of building consensus, assur- 
ing accurate measures, estimating costs, defining 
progress, addressing legal challenges and building pub- 
lic support. (This guide is jointly published with the 
National Center for Research on Evaluation, Standards 
and Student Testing.) 1997 (SI-97-3), 27 pp., $10.00 

So You Have Standards . . . Now What? 

With standards in place or being developed in nearly 
every state, next on the horizon are new assessments 
that measure progress toward achieving the standards 
and make it possible to target improvement efforts 
effectively. This guide provides tips and strategies on 
how to involve educators, the public and parents in 
deciding on new forms of assessment. Includes a list of 
concerns the public is raising about assessments and 
tips for meeting the policy and communications 
challenges ahead. 1997 (SI-97-2), 29 pp., $10.00 



Stages of Implementation 
of Standards-Led Education 

This wall chart describes the developmental 
stages of an education system implementing standards- 
based reforms. It describes how the system will shift in 
key areas such as public engagement, standards, assess- 
ment, curriculum and instruction, professional develop- 
ment, colleges and universities, and equity. This tool is 
especially effective in promoting dialogue and self- 
assessment among educators, administrators, policy- 
makers and the public. (It is the shortened version of the 
policy tool described below.) 1997 (SI-97-9), wall chart 
(24 X 22 inches), $5.00 

Stages of Implementation 
of Standards-Led Education 

This is similar to the wall chart described above 
but includes more in-depth information and three addi- 
tional policy areas — technology, system diagnosis and 
planning, and roles and responsibilities. This tool also is 
effectively used by groups to promote dialogue on 
where a school, district or state is on a continuum of 
change and provides a description of what the next level 
of implementation will look like. 1996 (SI-96-8), 16 
pp., $5.00 




To order one of the above publicationsy please call 303-299^3692. 
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